Synthetic Dataset: Prefix Caching Controls #183
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
dagrayvid
changed the title
sjmonson
force-pushed
the
feat/fixed_prefix
branch
2 times, most recently
from
9c12d0b to
94a4508
Compare
sjmonson
force-pushed
the
feat/fixed_prefix
branch
3 times, most recently
from
20b660b to
081cdab
Compare
Signed-off-by: Samuel Monson <smonson@redhat.com>
Signed-off-by: Samuel Monson <smonson@redhat.com>
Signed-off-by: Samuel Monson <smonson@redhat.com>
Co-authored-by: Mehul <MEHTMEHUL@GMAIL.COM> Co-authored-by: Samuel Monson <smonson@redhat.com> Signed-off-by: Samuel Monson <smonson@redhat.com>
Signed-off-by: Samuel Monson <smonson@redhat.com>
sjmonson
force-pushed
the
feat/fixed_prefix
branch
2 times, most recently
from
473d097 to
692589c
Compare
sjmonson
changed the title
There was a problem hiding this comment.
Pull Request Overview
This PR implements prefix caching controls for the synthetic dataset generator by adding configurable prefix buckets and ensuring unique prompts. The implementation allows control of token prefix cache rates through shared prefixes across samples while maintaining prompt uniqueness.
Key changes include:
- Added
PrefixBucketConfigfor configurable prefix generation with bucket weights, prefix counts, and token lengths - Modified prompt generation to include auto-incrementing unique prefixes and configurable shared prefixes
- Updated documentation to reflect the new prefix configuration options
Reviewed Changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/guidellm/dataset/synthetic.py | Core implementation of prefix bucket configuration and modified prompt generation logic |
| src/guidellm/dataset/init.py | Exported new PrefixBucketConfig class |
| tests/unit/dataset/test_synthetic.py | Comprehensive test suite for new prefix functionality |
| docs/datasets.md | Updated documentation with prefix_tokens parameter |
Comments suppressed due to low confidence (1)
docs/datasets.md:79
- The documentation describes a
prefix_tokensparameter, but the implementation usesprefix_bucketswith a more complex structure. This documentation appears to be outdated or incorrect for the current implementation.
- `prefix_tokens`: Number of tokens to share as a prefix across all prompts. Is additive to the prompt tokens distribution so each request is `prefix_tokens + prompt_tokens_sample()`. If unset, defaults to 0.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
sjmonson
force-pushed
the
feat/fixed_prefix
branch
from
558cc78 to
f254066
Compare
markurtz
approved these changes
Aug 20, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Work to allow control of token prefix cache rates with the synthetic data generator. Firstly adds an auto-incrementing single token prefix to ensure we never repeat the same prefix. Secondly adds controls for sharing a fixed prefix between samples.
Details
1. Ensure every prompt is unique
When generating a prompt, the first token is now taken from an iterator over the tokenizer vocab.
2. Add configurable prefix to simulate system prompt or other common token prefixes
Example usage:
Test Plan
pytest tests/unit/dataset)Related Issues
Use of AI
## WRITTEN BY AI ##)